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ABSTRACT 

Item components that might contribute to the 
difficulty of items on the Raven colored Progressive Matrices (CPM) 
and the Standard Progressive Matrices (SPM) were studied. Subjects 
providing responses to CPM items were 269 children aged 2 years 9 
months to 11 years 8 months, most of whom were referred for testing 
as potentially gifted. A second sample containing 147 seventh-grade 
students, drawn from J. K. Gallini's study n 1983. was used to 
assess the utility of the equation developed using the first item 
sample. CPM item characteristics were defined and rated. Rasch item 
difficulties were used as the dependent variable with misfitting 
items omitted. All 15 item characteristics were entered in a 
regression equation using forced entry (multiple "R" of 0.90) and 
stepwise entry (multiple "R" of 0.88). When the same predictors were 
used with SPM items, the multiple "R" was 0.69. The poorest 
prediction occurred for items containing characteristics (such as 
line thickness) that were not captured by the coding system. The best 
prediction occurred for items in which the orientation of the figure 
or options was a crucial feature. Results are discussed with regard 
to psychological processes and use of item characteristics to create 
new test items. Two sample test items are included, and two tables 
and an appendix present data on item difficulties. (SLD) 
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ABSTRACT 

The purpose of this study was to identify and test item 
characteristics that predict the difficulty of Raven's Colored and 
Standard Progressive Matrices items. Colored Progressive Matrices item 
characteristics were defined and rated; Rasch item difficulties were used 
as the dependent variable with misfitting items omitted. The multiple R 
was .90 (.88 using stepwise prediction). When the same predictors were 
used with Standard Progressive Matrices items, the multiple R was .69. 
Results are discussed with respect to psychological processes and to using 
item characteristics to create new test items. 



^ Paper presented at the Annual Meeting of the National Council on 
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INTRODUCTION 



Figural reasoning items such as those developed by Raven (1965, 
1985) have proven useful as nonverbal measures of fluid or analytic 
intelligence (»g») . While matrices tests have been used for years 
and while a number of researchers have examined the factor structure 
of these tests, little attention has b.*n paid to empirical 
examination of the item variations contributing to iter difficulty. 
The purpose of this study was to identify item components that might 
contribute to item difficulty and then to assess which components are 
predictive of empirical item difficulty. The prediction equation 
constructed was then used to predict difficulty of a second set of 

matrices items. 

The Raven Colored Progressive Matrices (CPM) and Standard 
Progressive Matrices (SPM) have be ^ used as measures of a unitary 
trait, although factor analyses suggest that 2-4 factors are 
necessary to explain item intercorrelations (e.g., Carlson & Jensen, 
1980; Schmidtke & Schaller, 1950). Analyses indicated that while a 
mult if actor solution was needed, item responses yielded adequate fit 
to a logistic model (the Rasch model: Green & Kluever, 1991). High 
internal consistency coefficients have also been reported (e.g., 
Court & Raven, 1982; Raven, Court & Raven, 1986). If all test items 
are reflective of a single latent trait, it should be possible to 
identify ways in which items differ that make some items 
systematically easier and others harder to answer. This process is 
termed component identification. Identification of components would 
prove useful in construction of additional test items and in 
furtherance of our understanding of the construct being measured. 
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Suggestions of bet?- method of analysis and of potential 
components may be obtained from previous work with Knox Cube Block 
Test items, paper-folding items, and progressive matrices items 
(Carpenter, Just, & Shell, 1990; Gallini, 1983; Green & Smith, 1987; 
Smith & Green, 1985; Ward & Fitzpatrick, 1973). Some of these item 
types differ from matrices items but all are non-verbal. Analyses of 
verbal analogies items have also been performed but these results 
seem less relevant to the analysis of nonverbal items. 

The number and complexity of components determine the difficulty 
of an item. Potential components relevant to matrices items defined 
by previous work include variables such as symmetry vs. assymmetry, 
vertical or horizontal vs. diagonal axes, straight lines vs. curved 
lines, size of cell attributes, number of dimensions of variation 
(e.g., different line widths, different shapes), proportionate vs. 
disproportionate change of size, shading, number of colors, 
intersection vs. union of dimensions, rotation of elements, and 
reflection of elements. Number, orientation, and figure type are the 
problem descriptors used by Carpenter et al. (1990) in their analysis 
of cognitive processes used by high and average performers on 
matrices items. More difficult problems involve more figural 
elements and/or more complex combination rules (Ward & Fitzpatrick, 
1973) . If multiple rules are needed in problem solution, cognitive 
management of rule construction and execution is taxed as well as the 
mental processes used to construct the rules. Carpenter et al. 
(1990) suggest that individual performance differences reflect 
abilities to generate and maintain problem solving goals in working 
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memory. Mulholland, Pellegrino, and Glaser (1980) found that errors 
and response times increased when the number of operations needed to 
solve geometric analogies increased. They also attributed this 
performance decrement to an increased burden on working memory 
created by the need to track more elements and transformations. 

The skill with which individuals process information is dependent 
on variables such as the kind(s) of cognitive processes involved, the 
nature of the content, the complexity of processing required, and 
one's previous experience with the task. Basic processing models 
involving variations of the input-process-output systems are common 
in the literature. One's experience with the content of the material 
is reflected in some processing models. Other models are more 
descriptive of the nature of the material and some models are more 
concerned with the cognitive complexity required to solve a problem. 

The Structure of Intellect (SOI) model (Guilford, 1959) lends 
itself as a framework to systematic analysis of the content and 
processes involved in solving matrices problems. Other relevant 
models are simultaneous-sequential processing models and Horn and 
Cattell's (1966) model of cognitive processing. Item difficulty and 
item characteristics are related in this paper to these models as 
indications of the cognitive processes involved, the nature of the 
content, and its complexity. 
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METHOD 



Subjects providing responses to CPM items were 269 2-9 to 11-8 
year-old children seen at a University Assessment Center for 
individual testing through 1989. The majority of these children were 
referred for testing as potentially gifted. Responses were fit to a 
linear logistic model using BICAL (Wright, Mead, & Bell, 1980). Two 
of the 36 CPM items misfit (both between and total fit >3.00) and so 
were dropped from subsequent analyses. Logit item difficulty 
obtained from BICAL was used as the dependent variable; the effective 
n for this study was 34. A second sample was obtained from data 
reported by Gallini (1983) who gave 30 of the 60 Standard Progressive 
Matrices items to 147 seventh-grade students from an urban middle 
school in the Southeast. Four of these 30 items were answered 
correctly by all students so logit difficulties of one less than the 
lowest logit value were assigned to these items. Three other items 
misfit and were dropped from the analysis leaving an effective n of 
27 for this sample. The second sample was used to assess the utility 
of the equation developed using the first item sample. 

Analyses were conducted using estimated Rasch item difficulties 
regressed on component frequencies. Item calibration using tho Rasch 
model provides a means to evaluate unidimensionality. Regression 
analyses are of limited value unless the dependent variable assesses 
a single trait (in this case, item difficulty). Modeling performance 
on a set of items that are not well-defined can be pointless. Thus, 
Rasch analysis with removal of misfitting items was used to refine 
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the dependent measure, to provide reasonable assurance of 
unidimensionality. Relationships among predictor variables was also 
assessed since the presence of multicollinearity leads to varied 
interpretation of results. 

CPM items consist of an item "stem" which contains a figural 
display with a missing piece, and 6 or 8 response options, one of 
which completes the figural pattern. Stem characteristics assessed 
were: vertical/horizontal orientation vs. other, symmetrical vs. 
asymmetrical, progression vs. not, number of dimensions in the 
pattern, straight lines vs. curved lines, number of lines or solids, 
density of design, and color vs. black and white. Number of 
dimensions and elements were coded a 0-3; all others were coded as 
0-1. Response option characteristics assessed were: number of 
distinct options (2-8), options contain progression, rotation, 
reflection (0-1 for each), number of directions of options (e.g., 
horizontal, vertical, diagonal; 1-3), number of elements in the 
design (1-3), and reversal between foreground and background (0-1). 
Characteristics were rated independently by the two authors; 
disagreements occurred for less clearly specified characteristics 
such as "density of design." Disagreements were either resolved or 
the category redefined. Figure 1 illustrates item component 
categorization for two hypothetical items. 
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EXAMPLES OF ITEM CHARACTERISTIC CODING 



Item 1 



in in 





Stem Variables: 

Vertical/horizontal orientation: O-yes 
Symmetrical : O-yes 
Progression: 0-no progression 

# dimensions in pattern: 1-pattern only 
Straight lines vs. curved: O-straight 

# elements: 1-lines 

Color vs. black/white: 1-black/white 

Option Variables: 

# distinct options: 5 

Options contain progression: 0-no 

rotation: 1-yes 
reflection: 1-yes (e.g., 1 and 5) 

# option directions: 3 (vertical, horizontal, diagonal) 
Reversal between foreground and background: 0-no 



Item 2 









Stem Variables: 

Vertical/horizontal orientation: 1-diagonal 
Symmetrical : 1-no 

Progression: 1-yes (angle of line separation increases) 

# dimensions in pattern: 2-size, orientation 
Straight lines vs. curved: O-straight 

# elements: 2 (lines, solids) 
Color vs. black/white: l-black/white 

Option Variables: 

# distinct options: 6 

Options contain progression: 1-yes (e.g., 2 and 6) 

rotation: 1-yes (e.g., 4 and 5) 
reflection: 1-yes. (e.g., 4 and 6) 

# option directions: 2 (vertical, diagonal) 
Reversal between foreground and background: 0-no 



RESULTS 



Table 1 presents the zero-order correlations of all 
characteristics with item difficulty for CPM items. Item 
characteristics were multicollinear. Table 2 presents significant 
(p<.05) inter-characteristic correlations for CPM items. CPM items 
were used as the basis for construction of a regression function. 
All 15 item characteristics were entered in a regression equation 
using two methods: forced entry and stepwise entry. With forced 
entry, the multiple B was .90. With stepwise regression, number of 
distinct options, reflection of one or more options, number of 
dimensions in the stem, and number of directions of options 
contributed significantly (p<.05) to prediction for a multiple g of 
.88. The B 2 adjusted for number of cases was .74. Actual and 
predicted item difficulties for CPM items are provided in Appendix A 
as is the standardized regression function. 

SPM item difficulty was predicted using the four characteristics 
identified as significant predictors of CPM item difficulty. The 
multiple B was .69. Actual and predicted item difficulties are 
listed in Appendix A. 

To assess the effect of nonlinearity, the squared and cubed 
values of components were added to prediction equations. This 
resulted in a small increase in prediction for CPM items (.88 to .91) 
and in a similar increase in prediction for SPM items (.69 to .71). 
When higher order terms were added, the variable representing number 
of options dropped out of the function. 



Table 1 

Correlation of Item Characteristics with Ttem Difficulty. 



Characteristic Cprrelatipn-CPM ££M_ 

Orientation 01 

Symmetry 61 

Progression 17 

Number of Dimensions 66 61 

Lines 22 

Lines-Solids 68 

Color 46 

Progression "* 18 

Rotation 28 

Reflection 34 31 

Number of Options 48 52 

Density ~ 02 

Number of Directions of Options 19 19 

Reversal of Foreground and Background 02 

Number of Elements 22 



Table 2 

Item Characteristic Intercorrelations-CPM 

C6 C7 C8 CI 3 C14 ~ 16_ _~ 

C2: Symmetry 78 76 53 

C4: Number of Dimensions 82 69 

C6: Lines-Solids 61 38 41 

C7: Color 45 
C9: Rotation 34 
C13: Density of Design -48 

C8: Progression; C14: Number of Directions of Options; C16: Number of 
Elements in Design 

Note: Only item characteristics that were significantly correlated (p<.05) 
are listed. 
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The two items from each test for which prediction was the poorest 
and best were reviewed. The poorest prediction occurred for items 
containing characteristics (such as line thickness) that were not 
captured by the coding system. The best prediction occurred for 
items in which orientation of figure or of options was a crucial 
feature. 

DISCUSSION 

Only tentative conclusions may be drawn from the results of this 
study. Some variables that were found to be significant predictors 
were correlated with other item characteristics. This collinearity 
makes determination of which unique characteristics predict item 
difficulty problematic. 

Orientation of options (rotation and reflection) were both 
significantly related to item difficulty with reflection somewhat 
more highly correlated. Many progressive matrices items involve 
orientation as well as design matching and number. Items tend to be 
more difficult when several options are identical in shape/number but 
are mirror images of each other. Rotation of an option away from a 
vertical/horizontal orientation does not seem to pose as difficult a 
problem as reversal in the same plane. For Raven's items, reflection 
seems to require a finer discrimination than rotation. Rotation for 
these items is likely to involve a disturbance to the orientation of 
the figure which is relatively easy to identify as incorrect . 
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Number of directions of options also was a significant predictor 
for CPN items. This variable also assessed spatial orientation. 
This variable did not significantly add to prediction for SPM items. 

Number of distinct options was a significant predictor for CPM 
items. All CPM items have six options and only two of the 30 items 
have less than six unique options. These are the second and third 
items on the test and are also among the easiest test items. SPM 
items have six or eight options; again, only two items have options 
that are not unique and these items are the easiest. This variable 
served to identify extremely easy items. 

Number of dimensions in the stem assessed the number of different 
figural elements that needed to be considered in problem solution. 
Possible elements were figural match, orientation, size, and number. 
Items varying on more than one dimension were more difficult. This 
variable was a measure of item complexity and was most predictive of 
difficulty for both CPM and SPM items. This variable is similar to 
number of transformations which previous researchers have also found 
to be predictive of item difficulty. 

While analysis of components does not describe the elementary 
mental processes necessary to problem solution, we propose congruity 
with certain processing models. Processing models may include 
components such as the ability to attend, to encode information, 
short- and long-term retention, the ability to retrieve infromation 
from storage, certain cognitive processing skills, and an output 
component. Task, content, and situational characteristics are 
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important factors to consider in analyzing a specific processing 
requirement . 

Among models of cognitive processing, the Horn and Cattell (1966) 
theory includes fluid and crystallized ability representing the 
processing of new versus more familiar material. Success with 
components of the CPM items probably places a high premium on fluid 
abilities since solving matrices tasks is not common. At best, the 
analysis leading to the correct choice for stimulus complex ".on may 
parallel certain common everyday discrimination tasks but the CPM 
content is certainly unique. 

Analysis of this fluid ability for solving the CPM problems can 
be in the context of a simultaneous and sequential processing scheme 
proposed by several psychologists (Das, Kirby, & Jarman, 1975; Luria, 
1973). Solution of the CPM problems seems to require good 
simultaneous processing ability of a gestalt-like configuration with 
differing components contributing to processing load as identified in 
this study. 

A model displaying greater detail of processing skills is 
Guilford's Structure of Intellect (SOI) Model (1967). The Raven 
designs probably represent the Figural Content as defined in the 
model. Among the five Operations described in the model, Cognition 
and Evaluation abilities appear to be most representative of the kind 
of processing required to solve the CPM problems. The Products of 
this model ranging from unitary components to complex abstracting 
abilities requiring the reconfiguration of the stimulus material are 
also evident in the Raven problem-solving requirements. 
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Both the SOI model and the simultaneous processing-sequential 
processing model have published tests based on those concepts. The 
value of the component analysis in this study is the opportunity it 
presents for construction of items that utilize component 
characteristics in revisions of these tests to tap a defined range of 
processing abilities. This has implications for test construction 
and for test interpretation where manuals could provide guidelines 
for this kind of interpretation. 

There are several limitations of this study. First, all features 
of items were not assessed nor were problem-solving process variables 
assessed. Only fairly obvious observable features were included in 
the analysis. Second, the regression analysis assumes that a linear 
combination of variables predicts item difficulty. Even if the 
appropriate components have been identified, they may relate to other 
components and to item difficulty in a nonlinear fashion. Third, 
individual children may use different strategies in solving matrices 
items. The analyses performed implicitly assume that all subjects 
use similar strategies and attend to similar aspects of problems. 
This assumption may be overly simplistic. Finally, the size of the 
samples used in calibrating items were smaller than those desirable 
to establish highly reliable item statistics. 
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APPENDIX A 

Actual and Predicted Item Difficulties for CPM and SPM Items 



Difficulty Difficulty 



CPM Item 
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.43 
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-2 .88 
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.21 


.15 


RC5 


1.56 


1.42 
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.21 
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1.37 


.22 


RAB6 


1.04 


-1.87 


RD1 


-3.44 


-1.69 


RAB7 


.54 
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RD2 


-2.21 


-1.69 


RAB8 


1.88 


2.11 


RD3 


-.59 
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1.74 


.15 
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Zy ■ . 17 (foptions) + .43 (reflection) + .75(#dimensions) + 
. 29 (# option directions) + e 
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